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In the fall of 2007 a major public research university undertook a review of basic 
skills and placement in two departments. The process of review and renewal in the two 
programs differed substantially, and divergent processes employed and outcomes gained 
may serve as an object lesson for the role of assessment within the process of institutional 
change. As in most social science case studies, there is more than one possible reading of 
the story. This narrative will focus on a specific set of differences from which lessons can 
be drawn. It might well be true that the less successful case could not have ended 
otherwise because of the underlying power structure. Still, by contrasting the cases, 
management lessons may be learned and those pursuing effective institutional change 
may find more broadly applicable ideas. 

The framework for understanding the management difference is neatly captured 
by the ‘red ocean’ and ‘blue ocean’ strategies described by W. Chan Kim and Renee 
Mauborgne in ‘Blue Ocean Strategy’ (Kim & Mauborgne, 2004). In the original, the 
terms refer to alternative market strategies. Red ocean means competition in a market 
defined by a zero sum game. The gain of one party entails loss for another. This is 
typical for any situation where the limits and rules are long established. For example, if a 
company is selling automobiles they may pursue a strategy of selling the usual car at a 
lower price. Whatever market share they gain with this competitive strategy will entail a 
loss for other participants. 

An alternative, less frequently used strategy is the blue ocean strategy which 
means developing new limits or rules which change the market in such a way that it is no 
longer a zero sum game. For example, a competitor in the automobile business might 
move toward radical innovation by developing a new type of automobile, perhaps a first 
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of its kind hydrogen car. This new technology would offer consumers a radically 
different alternative and open a new market with new rules. Although there is still 
competition between the old and the new technologies both can exist in the market 
without direct competition in price. The question is whether a new market for hydrogen 
cars will emerge. 

The contrast between blue and red ocean strategies in business is well established. 
What it means for organizational change in the business of higher education is perhaps 
best demonstrated through competing case studies. 

The Red Ocean Strategy of Change 

In summer 2007 the administration of a senior public research university 
requested a study of basic skills courses in one department. The impression had grown on 
people for some years that either the admissions criteria for students were too low or the 
remedial test was placing students inappropriately because the number of entering 
students in that department’s basic skills programs was expanding at an alarming rate. 

To address this question a hardnosed assessment specialist developed a detailed 
quantitative study of the remedial exam and student placement rates. Comparing 
placement rates to admissions criteria the assessor produced a compelling case for the 
inappropriateness of the exam. The report showed no significant link between the 
placement exam and performance in the department’s courses. It also showed no 
significant correlation between the exam and admissions criteria like class rank or SAT 
scores. In conclusion the report advocated a specific set of changes designed to address 
identified problems. Among the proposed changes was relying on existing admissions 
criteria more heavily and deemphasizing certain aspects of the placement exam. After 




Institutional Change through Assessment 4 



producing the report the assessor bypassed review by the department faculty, believing 
they would suppress it and took the report directly to senior staff. 

Upon reviewing the report, senior staff confronted the department and insisted on 
change. In a compromise to the department, however, the assessor was reprimanded and 
the department was placed in charge of its own reform, including all future assessment. 
Within one year a new exam had been selected and the entire remedial system was 
revised. The new exam used national benchmarks to place approximately the same 
number of students in remedial courses. All future assessment was handled by the 
department and demonstrated, conclusively, the effectiveness of the placement program 
and the remedial course structure. Change was implemented, but it bore no resemblance 
to the changes proposed by the assessor, nor were the assessor’s proposed changes ever 
seriously considered. Moreover, the changes did not address the fundamental concerns 
raised by the assessor. 

This was a classic red ocean approach. The assessor naively sought to force 
change on a department through an appeal to power. This would have dramatically 
asserted the authority of the assessment office while undermining the academic 
department concerned. Effective change in higher education, however, seldom comes 
about through the assertion of power. 

The Blue Ocean Strategy of Change 

Parallel with the assessment which employed the red ocean strategy, a veteran 
faculty member was asked to lead a review of a different remedial program. The same 
hardnosed assessor had concluded in the original report that this other remedial program 
did not suffer the same level of disconnect between student needs and student placement. 
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Nevertheless, the veteran faculty member seized the opportunity to use assessment as a 
vehicle for implementing change. The method he employed took more time and acted 
more indirectly, but achieved a more dramatic set of changes with a markedly smaller 
cost. In the end, there was no zero sum game and opponents of change appeared as 
recalcitrant defenders of an antiquated regimen. 

The process for achieving this blue ocean change can be seen as following a six 
step process: 1) Problem articulation; 2) Organic solution emergence; 3) Change 
implementation; 4) Assessing change; 5) Refining change; 6) Ongoing assessment. 

Problem Articulation 

In December 2007 the faculty member produced an academic sounding memo 
outlining a set of criteria for assessment and detailing a series of studies identifying the 
problem. He then proceeded to argue for a further set of studies and a specific agenda for 
change. This was followed by the rapid intervention of senior staff to insist on a new 
remedial testing strategy. A further memo in January 2008 accepted the proposed new 
remedial exam and outlined a detailed evaluation protocol for it. The memo included a 
detailed sample study of the new exam to calibrate equivalent scores to the old placement 
exam. It further proposed a series of validation studies to test the appropriateness of the 
new placement procedure. The memo included a detailed explanation of costs and 
benefits associated with the new placement process and the evaluation studies 
accompanying it. 

An extensive quantitative addendum, ironically prepared in collaboration with the 
hardnosed assessor from the red ocean case study, followed in March, with additional 
material provided in April. These reports included a detailed power point presentation for 
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the dean and senior staff and carefully thanked all stakeholders for their collaboration in 
the evaluation process, including those stakeholders most fervently opposed to it. It also 
included extensive textual explanations for consideration of those who did not hear the 
original presentation. 

The memos and power point presentation, detailed results of calibration studies 
and provided quantitative reassurance about the validity of the new system, including 
tests of gender and cultural bias. It also laid the groundwork for an alternative should the 
proposed exam not perform well in its first year of full implementation. The preparation 
of this alternative, would become critical to success a year later. 

Organic Solution Emergence 

Whether by design or fortuitous accident, throughout the first stage analyses, a 
latent alternative operated. This was based entirely on a refined analysis of existing 
admissions material and did not rely on the proposed placement exam. In effect, the 
existence of this alternative in many analyses laid the groundwork for a ‘fall back’ 
position with which the assessment team was increasingly comfortable. When the initial 
change proved less than ideal, an alternative was already at hand and seemingly radical 
change could be implemented without surprise. As all organic change, however, the 
emergence of this alternative is not neatly placed in a temporal framework and we will 
return to this in each of the following steps. 

Change Implementation 

Throughout spring and summer 2008, the new basic skills process was 
implemented according to the design proposed by the assessing faculty member. This 
change involved the laborious development of new systems and procedures across a 
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range of competing offices. Students were given a placement exam and placed in courses 
according to the calibrated cut scores developed in March and April. This process 
involved a wide range of department faculty working closely with members of the 
freshmen studies office and the department of institutional research and planning. 

Why, you might wonder, was the office of institutional research involved in the 
implementation of this change? The answer is that change should not be implemented 
without clear preparation for future assessment. Properly planned, future assessment of 
change requires careful preparation. The appropriate data must be collected and stored, 
including details which may not be necessary to the placement process itself. As well, the 
department needed frequent consultation with institutional research colleagues on 
experimental design and statistical analysis. Without such collaboration, the department 
would have operated without technical information that was critical to the success of the 
experiment. 

Finally, it is critical to recognize that change itself builds the capacity for change. 
Once an organization has been successfully dragged out of the standard operating 
procedures the potential for further change is increased. (Blau, 1971) The changes 
proposed and implemented to transform the remedial placement procedure engendered 
opposition, particularly from those accustomed to 13 years of the previously operating 
system. Once the familiar methods had been overturned the natural inertia of institutions 
favoring routine no longer precludes a broader range of alternatives. It was the reduction 
of this natural inertia which allowed the organic emergence of a different solution. 
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Assessment should be built into the plan and prepared in advance. The existence 
of a prior plan reduces the danger of objective driven assessment. Once the change is 
implemented it may be favored because it requires less work or derided because the 
natural inertia of institutions is to return to a previously established state. 

In this case, the assessment strategy was carefully mapped out in advance and was 
designed, if any thing, to vindicate the wisdom of the change. There was, conveniently, 
no possibility of return to the previous system because the preliminary work of the 
December and January memos had thoroughly delegitimized it. 

The assessing faculty member, again working with the now thoroughly 
reprimanded hardnosed assessor, produced a detailed analysis of the basic skills system 
in January 2009, within weeks of the final semester grades. This assessment report 
provided an opportunity to not only assess the change but move the change even further 
forward toward an organic solution. 

The assessment employed extensive statistical studies as back up, but it opened 
with two key claims and a proposal. The first claim asserted the wisdom of the change: 
“The placement method... was very effective in assigning students to the correct first-year 
writing courses.” The second bluntly outlined newly discovered risks and costs associated 
with the new system. The report then offered a radical proposal to abolish the placement 
system entirely in favor of specific, already existing, admissions criteria. 

To the astonishment of the hardnosed assessor who had failed so spectacularly in 
the red ocean approach, the proposal was accepted with only limited opposition. 



Ironically, a carefully couched similar proposal offered by the assessor in the red ocean 
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approach was met with ridicule, although the statistical support was every bit as 
compelling. 

Why were the three comments offered in the January 2009 memo readily accepted 
while the red ocean style report failed? 

This touches on the thorny problem of establishing an idea’s credibility. (Shapin 
1994) The first claim reaffirms the wisdom of the faculty assessor’s initially proposed 
changes. If the opening claim had read as an admission of error it would have 
undermined any following proposal. The second claim is a brilliant twist of institutional 
jujitsu. It plays on the fears people have about the new system, which has already been 
found to be successful, it also takes the natural inclination to reduce labor and casts it as 
the course of wisdom. Finally, the claim uses the uncertainty of weak institutional 
procedures regarding the remedial process to establish a fundamental critique of the 
system. 

It is important to recognize that both claims were informed by a perspective of 
logic and argument based on a system of persuasion articulated by Stephen Toulimn. 
(Mislevy, 2006, 2007). As adopters of blue ocean strategies, assessment specialists 
should come to view validity as a series of integrated judgments (Messick, 1989; AERA, 
APA, & NCME, 1999; Brennan, 2006), context- specific arguments (Mislevy 2006, 2007) 
and procedurally-dependent observations resembling science itself (Embretson, 2007). 
Such a new orientation of offering validity arguments is congruent with the new 
framework offered by a blue ocean philosophy of opportunity development. 

It was critical that this memo appeared in January immediately before people 
would begin implementing the remedial system for the next freshmen class. They were 
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already worrying about a new set of problems and how they would adjust relatively new 
procedures to address them. There was no old system to which they could return and they 
had a natural disinclination to undertake hard labor for little gain if there was a logical 
alternative. With those preconditions, the stage was set to grant credibility to the detailed 
statistical study which supported a new change doing away with placement testing, 
except in certain ambiguous cases, and relying instead on existing admissions criteria. 
Here was a final radical proposal which was readily adopted. 

Refining Change 

By itself, change in the assessment process is undesirable. Our blue ocean goal 
therefore had to be a shift from a static model (“Thank heavens that project is done!”) to 
an emerging model (“Now that we know the results, how can we improve our efforts?”) 

Citing recent work in validation by M. T. Kane published that very year, the 
December 2007 memo argued that all shareholders must shift from a product-oriented 
notion of validity to a process-oriented conceptualization of validation. Completing that 
unbroken cycle of validation a year later, a December 2008 memo reminded senior NJIT 
administrators, as well as faculty members, that the university must reject a product- 
oriented, mechanistic assessment model (termed the “shotgun approach” in the memo) 
and adopt an experimental model (embracing validation, curricular alignment, 
technological support, and cost estimation). The content of these two memos, a year 
apart, documents that, from the start, the blue ocean strategy incorporated a set of 
strategic moves identified by Kim and Mauborgne (2007, p. 81). We had created an 
uncontested space for the assessment through strategic briefings; we had, thus, made the 
competition — the red ocean strategy described above — irrelevant in its static view. We 
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had created and captured a new demand for the kind of information only the blue ocean 
was capable of providing. We had deliberately problematized the traditional value/cost 
tradeoff by offering a model process model that embraced the complexities of 
assessment. And we aligned all shareholders, administrators and faculty alike, within a 
new system that pursued both differentiation of assessment results and the pursuit of low 
costs as new assessment technologies arose. 

The impact of our process of refining change is evident is our new project: an 
analysis of automated assessment. We have just completed partnership with one of the 
nation’s leading non-profit educational research corporations to experiment with 
connections between instruction and assessment. Once again, student performance will be 
examined through a new assessment tool. In the fall of 2009, the department embracing 
the blue ocean strategy will incorporate the product of the educational research 
corporation into its own locally-based instructional strategies to study differentiation 
results of student achievement and potential cost reduction. The philosophy of red ocean 
would have rejected this new assessment experiment. The red ocean captains would have 
noted that automated assessment already had too much competition for an existing 
market, that the competition would be unbeatable, that the existing demand could not be 
further exploited, that the value/cost tradeoff could not be met, and that while student 
differentiation might be met, the goal of low cost could not. 

For the red ocean captains, the voyage was safely over. For the blue ocean 
captains, the voyage was about to begin again. By adopting a process of refining and re- 
defining change, the blue ocean assessment developers had embraced a different system 
of logic, one that incorporated process into the very fabric of its decisions. 
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As a final piece of reassurance the memo outlines ongoing studies and assessment 
to determine whether the new admissions criteria based system performs well. After 
having established the basic credibility of the proposal the assessor reassures the 
institutional audience that this only requires a single year of experimental 
implementation. Next year, if it proves in effective we will know it and can undertake the 
additional work required to implement the system which worked well this year. 

Conclusions 

This case study contrasts two efforts to implement change in a university setting. 
The first adopts a zero-sum approach — called a red ocean strategy — which fails 
miserably, tainting the advocate as unreliable and hasty. Although the red ocean proposal 
relies on strong statistical evidence it is summarily dismissed and given no credibility 
because it fails to lay the groundwork for its own believability. Adopting the proposal 
would only cost stakeholders without bringing any credible benefit. 

The second parallel proposal adopted a blue ocean strategy where change was 
introduced in stages and assessed along the way. It systematically prepares stakeholders 
for a change in which they can ultimately believe because it appears credible and 
beneficial. This approach successfully implemented substantial change by initially 
proposing an alternative placement test and then settling on a radical proposal which 
could not have been proposed in at the launch of the process. Ironically, the final 
proposal echoes a change originally proposed in the red-ocean style assessment where it 
was immediately dismissed. The strategy by which information is presented, it appears, is 



as important as the information itself — a statement that would suggest that we are now 
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entering firmly into a post- structural period in institutional research. The metaphor of a 
system — classificatory, measurable, accountable — once uniformly adopted in America, 
as David Harvey (1989) has shown, may well be behind us. As Thornton and Jaeger 
(2007) have found, organizational cultural studies, embracing localism and the distinct 
patterns therein, may he of great help in understanding how arguments may be advanced 
within specific institutional sites. Ahead of us is an era of contingency in which 
everything depends not on system but, rather, on context. 

Other instrumental analyses for the success and failure of these cases could be 
proposed, yet that is not the point here. These two cases illustrate fundamentally different 
approaches to institutional change, and they produce radically different results for 
predictable reasons. By drawing lessons from these cases, we may improve the 
effectiveness of our efforts to use assessment data when implementing institutional 
change. The six steps of the blue ocean strategy — 1) Problem articulation; 2) Organic 
solution emergence; 3) Change implementation; 4) Assessing change; 5) Refining 
change; 6) Ongoing assessment — offer steps for the reflective institutional researcher to 
follow when undertaking the often dangerous task which is the essence of our mandate. 




Institutional Change through Assessment 14 



References 

American Educational Research Association, American Psychological Association, & 
National Council on Measurement in Education. (1999). Standards for 
Educational and Psychological Testing. Washington, DC: American 
Psychological Association. 

Blau, P. M. & Meyer, M. W. (1971). Bureaucracy in modem society. 2 nd ed. New York: 
Random House 1971. 

Brennan, R. L. (2006). Perspectives on the evolution and future of educational 

measurement. In R. L. Brennan (Ed.). Educational measurement (4th edition, 1- 
16.). Westport, CT: Praeger. 

Harvey, D. (1989). The condition of postmodernity: An enquiry into the origins of 
cultural change. Cambridge and Oxford: Blackwell. 

Kane, M. T. (2006) Validation. In R. L. Brennan (Ed.), Educational Measurement (4th 
ed., pp 17-64. Westport, CT: American Council on Education and Praeger. 

Kim, W. C. & Mauborgne, (2004) R. Blue ocean strategy. Harvard Business Review , 
Oct. 2004. 

Messick, S. (1989). Validity. In R. L. Linn (Ed.). Educational measurement (3rd edition, 
13-103). New York: American Council on Education/Macmillan. 

Mislevy, R. J. Brennan, R. L. (2006). Cognitive psychology and educational assessment. 
In R. L. Brennan (Ed.). Educational measurement (4th edition, 257-305.). 
Westport, CT: Praeger. 

Mislevy, R. J. (2007). Validity by design. Educational Researcher 36, 463-469. 




Institutional Change through Assessment 15 



Shapin, S. (1994). A social history of truth: Civility and science in seventeenth-century 
England . University of Chicago Press. 

Thornton, C. H. & Jaeger, A. J. (2007). A new content for understanding civic 

responsibility: Relating culture to action at a research university. Research in 
Higher Education, 48, 993-1020. 




